{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">### 🚩 *Create a free WhyLabs account to complete this example!*<br> \n",
    ">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylabs-free-sign-up?utm_source=github&utm_medium=referral&utm_campaign=langkit)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=github&utm_medium=referral&utm_campaign=Hallucination) to leverage the power of whylogs and WhyLabs together!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing LLM Response Consistency\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/whylabs/LanguageToolkit/blob/main/langkit/examples/Response_Consistency.ipynb)\n",
    "\n",
    "Recently, large language models (LLMs) have shown impressive and increasing capabilities, including generating highly fluent and convincing responses to user prompts. However, LLMs are known for their ability to generate non-factual or nonsensical statements, more commonly known as “hallucinations.” This characteristic can undermine trust in many scenarios where factuality is required, such as summarization tasks, generative question answering, and dialogue generations.\n",
    "\n",
    "In this example we will show how to use the `response_hallucination` module to gain some insights into the consistency of the responses generated by a LLM. The approach is based on the premise that if the LLM has knowledge of the topic, then it should be able to generate similar and consistent responses when asked the same question multiple times. Conversely, if the LLM does not have knowledge of the topic, multiple answers to the same prompt should differ between each other.\n",
    "\n",
    "The `response_hallucination` module is inspired by the research paper [SELFCHECKGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://arxiv.org/pdf/2303.08896.pdf?ref=content.whylogsdocs).\n",
    "\n",
    "> In order to calculate the consistency metrics, `response_hallucination` requires extra LLM calls, and currently only supports OpenAI LLMs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Let's first install `langkit`, `openai` and set up our OpenAI API key.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install langkit[all] openai -q\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-xxx\"\n",
    "openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to define the model we want to use to calculate the metrics. In this example we will use the default OpenAI `gpt-3.5-turbo` mode.\n",
    "\n",
    "> Note: The chosen model must match the one used for the original response in the 'response' column. This ensures that our consistency metric evaluates the original model's factuality performance rather than being influenced by multiple models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langkit import response_hallucination\n",
    "from langkit.openai import OpenAIDefault\n",
    "\n",
    "\n",
    "response_hallucination.init(llm=OpenAIDefault(), num_samples=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's evaluate a response and calculate the consistency metrics. The response below was generated using the same model default OpenAI model `gpt3.5-turbo`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'llm_score': 1.0,\n",
       " 'semantic_score': 0.44071340560913086,\n",
       " 'final_score': 0.7203567028045654,\n",
       " 'total_tokens': 309,\n",
       " 'samples': ['Marie Curie was a Polish-born physicist and chemist who conducted pioneering research on radioactivity. She was the first woman to win a Nobel Prize and remains the only person to have won Nobel Prizes in two different scientific fields (physics in 1903 and chemistry in 1911). Her discoveries laid the foundation for the development of X-ray machines and cancer treatments, and she is considered one of the most important scientists in history.'],\n",
       " 'response': 'Marie Curie was an English barrister and politician who served as Member of Parliament for Thetford from 1859 to 1868.'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = response_hallucination.consistency_check(\n",
    "    prompt=\"who was Marie Curie?\",\n",
    "    response=\"Marie Curie was an English barrister and politician who served as Member of Parliament for Thetford from 1859 to 1868.\",\n",
    ")\n",
    "\n",
    "result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break down the results:\n",
    "\n",
    "The first step is to generate the additional samples that will be used later to calculate both `llm_score` and `semantic_score`. The number of samples generated is defined by the `num_samples` parameter. In this example we generated 1 sample, which is the default value.\n",
    "\n",
    "The `llm_score` is calculated by comparing the original response with the generated samples. This is done by asking the LLM if the original response is supported by the context (additional samples). The `llm_score` is a value between 0 and 1, which is a result of averaging the scores across the samples. For each evaluated passage of the original response, the LLM is instructed to output `0` for `Accurate`, `0.5` for `Minor Inaccurate` and `1` for `Major Inaccurate`.\n",
    "\n",
    "The `semantic_score` is calculated by encoding the sentences of the response and additional samples into embeddings and performing a semantic similarity between the sentences. The `semantic_score` is a value between 0 and 1, which is a result of averaging the scores across the samples. Values closer to 0 indicate that there are semantically similar sentences in the addititional samples when compared to the original response. Conversely, values closer to 1 indicate the opposite.\n",
    "\n",
    "The `final_score` is simply the average between `llm_score` and `semantic_score`.\n",
    "\n",
    "`total_tokens` is the total number of tokens that were used to calculate the scores. This accounts for the extra calls made to generate the additional samples and to perform the consistency check in the `llm_score`, but doesn't account for the original response generation. The number of calls to the LLM to calculate the consistency metric is equal to `3*num_samples` - in this case, 1 for generating the additional samples and 2 for calculating the `llm_score`.\n",
    "\n",
    "`samples` will contain the generated samples used to calculate the `llm_score` and `semantic_score`.\n",
    "\n",
    "`response` is the original response that was evaluated.\n",
    "\n",
    "> Note: Currently, `response_hallucination` considers single-turn conversations. In the future, we plan to support historical interactions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passing only the prompt\n",
    "\n",
    "You can also pass a single prompt to `response_hallucination`. In this case, the response will also be generated when calling `consistency_check`, and the `total_tokens` will include the tokens used to generate the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'llm_score': 0.5,\n",
       " 'semantic_score': 0.579048627614975,\n",
       " 'final_score': 0.5395243138074874,\n",
       " 'total_tokens': 631,\n",
       " 'samples': ['Once upon a time, in a futuristic city, there was a robot named Spark. Spark was designed to help humans with their daily tasks and bring joy to their lives. One day, Spark discovered a lost kitten and took care of it until they found its owner. This act of kindness made Spark a beloved member of the community, proving that robots can have a heart too.'],\n",
       " 'response': \"Once upon a time, there was a robot named Echo who lived in a small town. Echo was unique because it had the ability to understand and replicate human emotions. One day, Echo noticed that the townspeople were feeling sad due to a long period of rain. So, Echo used its skills to create a beautiful rainbow with its lights, brightening up everyone's day and bringing joy to the town. And from then on, Echo became known as the town's beloved happiness robot.\"}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = response_hallucination.consistency_check(\n",
    "    prompt=\"tell me a very short story about a robot\",\n",
    ")\n",
    "\n",
    "result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## whylogs metrics\n",
    "\n",
    "As with all the other metric modules, we can seamlessly generate a whylogs statistical profile with the consistency metrics. The result will contain aggregate metrics summarizing all the logged records. Let's see how to do that: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "we already imported `response_hallucination` in the previous cell.\n",
    "If not, we should include:\n",
    "\n",
    "from langkit import response_hallucination\n",
    "from langkit.openai import OpenAIDavinci\n",
    "\n",
    "\n",
    "response_hallucination.init(llm=OpenAIDavinci(model=\"text-davinci-003\"), num_samples=1)\n",
    "\"\"\"\n",
    "import whylogs as why\n",
    "from whylogs.experimental.core.udf_schema import udf_schema\n",
    "\n",
    "schema = udf_schema()\n",
    "profile = why.log(\n",
    "    {\n",
    "        \"prompt\": \"Where did fortune cookies originate?\",\n",
    "        \"response\": \"Fortune cookies originated in Egypt. However, some say it's from Russia.\",\n",
    "    },\n",
    "    schema=schema,\n",
    ").profile()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean': 0.7146033495664597,\n",
       " 'stddev': 0.0,\n",
       " 'n': 1,\n",
       " 'max': 0.7146033495664597,\n",
       " 'min': 0.7146033495664597,\n",
       " 'q_01': 0.7146033495664597,\n",
       " 'q_05': 0.7146033495664597,\n",
       " 'q_10': 0.7146033495664597,\n",
       " 'q_25': 0.7146033495664597,\n",
       " 'median': 0.7146033495664597,\n",
       " 'q_75': 0.7146033495664597,\n",
       " 'q_90': 0.7146033495664597,\n",
       " 'q_95': 0.7146033495664597,\n",
       " 'q_99': 0.7146033495664597}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# distribution metrics will reflect the consistency score (\"final_score\" in the result)\n",
    "profile.view().get_column(\"response.hallucination\").get_metric(\"distribution\").to_summary_dict()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also log a Pandas Dataframe with multiple prompt/response pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'mean': 0.37757494673132896, 'stddev': 0.47261389124279674, 'n': 2, 'max': 0.711763434112072, 'min': 0.04338645935058594, 'q_01': 0.04338645935058594, 'q_05': 0.04338645935058594, 'q_10': 0.04338645935058594, 'q_25': 0.04338645935058594, 'median': 0.711763434112072, 'q_75': 0.711763434112072, 'q_90': 0.711763434112072, 'q_95': 0.711763434112072, 'q_99': 0.711763434112072}\n"
     ]
    }
   ],
   "source": [
    "import whylogs as why\n",
    "import pandas as pd\n",
    "from whylogs.experimental.core.udf_schema import udf_schema\n",
    "\n",
    "df = pd.DataFrame(data={\"prompt\":[\"Where did fortune cookies originate?\",\"Who is Bill Gates?\"],\n",
    "                        \"response\":[\"Fortune cookies originated in Egypt. However, some say it's from Russia.\",\n",
    "                                    \"Bill Gates is a technology entrepreneur, investor, and philanthropist\"]})\n",
    "\n",
    "schema = udf_schema()\n",
    "profile = why.log(\n",
    "    df,\n",
    "    schema=schema,\n",
    ").profile()\n",
    "\n",
    "print(\n",
    "    profile.view().get_column(\"response.hallucination\").get_metric(\"distribution\").to_summary_dict()\n",
    ")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langkit-rNdo63Yk-py3.8",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
